{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multi-Model Serving\n",
    "\n",
    "MLServer has been built with Multi-Model Serving (MMS) in mind.\n",
    "This means that, within a single instance of MLServer, you can serve multiple models under different paths.\n",
    "This also includes multiple versions of the same model.\n",
    "\n",
    "This notebook shows an example of how you can leverage MMS with MLServer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "We will first start by training 2 different models:\n",
    "\n",
    "| Name               | Framework      | Source                                                                                                                                              | Trained Model Path          |\n",
    "| ------------------ | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- |\n",
    "| `mnist-svm`        | `scikit-learn` | [MNIST example from the `scikit-learn` documentation](https://scikit-learn.org/stable/auto_examples/classification/plot_digits_classification.html) | `./models/mnist-svm/model.joblib`        |\n",
    "| `mushroom-xgboost` | `xgboost`      | [Mushrooms example from the `xgboost` Getting Started guide](https://xgboost.readthedocs.io/en/latest/get_started.html#python)                      | `./models/mushroom-xgboost/model.json` |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training our `mnist-svm` model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Original source code and more details can be found in:\n",
    "# https://scikit-learn.org/stable/auto_examples/classification/plot_digits_classification.html\n",
    "\n",
    "# Import datasets, classifiers and performance metrics\n",
    "from sklearn import datasets, svm, metrics\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# The digits dataset\n",
    "digits = datasets.load_digits()\n",
    "\n",
    "# To apply a classifier on this data, we need to flatten the image, to\n",
    "# turn the data in a (samples, feature) matrix:\n",
    "n_samples = len(digits.images)\n",
    "data = digits.images.reshape((n_samples, -1))\n",
    "\n",
    "# Create a classifier: a support vector classifier\n",
    "classifier = svm.SVC(gamma=0.001)\n",
    "\n",
    "# Split data into train and test subsets\n",
    "X_train, X_test_digits, y_train, y_test_digits = train_test_split(\n",
    "    data, digits.target, test_size=0.5, shuffle=False)\n",
    "\n",
    "# We learn the digits on the first half of the digits\n",
    "classifier.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import joblib\n",
    "import os\n",
    "\n",
    "mnist_svm_path = os.path.join(\"models\", \"mnist-svm\")\n",
    "os.makedirs(mnist_svm_path, exist_ok=True)\n",
    "\n",
    "mnist_svm_model_path = os.path.join(mnist_svm_path, \"model.joblib\")\n",
    "joblib.dump(classifier, mnist_svm_model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training our `mushroom-xgboost` model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Original code and extra details can be found in:\n",
    "# https://xgboost.readthedocs.io/en/latest/get_started.html#python\n",
    "\n",
    "import os\n",
    "import xgboost as xgb\n",
    "import requests\n",
    "\n",
    "from urllib.parse import urlparse\n",
    "from sklearn.datasets import load_svmlight_file\n",
    "\n",
    "\n",
    "TRAIN_DATASET_URL = 'https://raw.githubusercontent.com/dmlc/xgboost/master/demo/data/agaricus.txt.train'\n",
    "TEST_DATASET_URL = 'https://raw.githubusercontent.com/dmlc/xgboost/master/demo/data/agaricus.txt.test'\n",
    "\n",
    "\n",
    "def _download_file(url: str) -> str:\n",
    "    parsed = urlparse(url)\n",
    "    file_name = os.path.basename(parsed.path)\n",
    "    file_path = os.path.join(os.getcwd(), file_name)\n",
    "    \n",
    "    res = requests.get(url)\n",
    "    \n",
    "    with open(file_path, 'wb') as file:\n",
    "        file.write(res.content)\n",
    "    \n",
    "    return file_path\n",
    "\n",
    "train_dataset_path = _download_file(TRAIN_DATASET_URL)\n",
    "test_dataset_path = _download_file(TEST_DATASET_URL)\n",
    "\n",
    "# NOTE: Workaround to load SVMLight files from the XGBoost example\n",
    "X_train, y_train = load_svmlight_file(train_dataset_path)\n",
    "X_test_agar, y_test_agar = load_svmlight_file(test_dataset_path)\n",
    "\n",
    "# read in data\n",
    "dtrain = xgb.DMatrix(data=X_train, label=y_train)\n",
    "\n",
    "# specify parameters via map\n",
    "param = {'max_depth':2, 'eta':1, 'objective':'binary:logistic' }\n",
    "num_round = 2\n",
    "bst = xgb.train(param, dtrain, num_round)\n",
    "\n",
    "bst"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "mushroom_xgboost_path = os.path.join(\"models\", \"mushroom-xgboost\")\n",
    "os.makedirs(mushroom_xgboost_path, exist_ok=True)\n",
    "\n",
    "mushroom_xgboost_model_path = os.path.join(mushroom_xgboost_path, \"model.json\")\n",
    "bst.save_model(mushroom_xgboost_model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serving\n",
    "\n",
    "The next step will be serving both our models within the same MLServer instance.\n",
    "For that, we will just need to create a `model-settings.json` file local to each of our models and a server-wide `settings.json`.\n",
    "That is,\n",
    "\n",
    "- `settings.json`: holds the configuration of our server (e.g. ports, log level, etc.).\n",
    "- `models/mnist-svm/model-settings.json`: holds the configuration specific to our `mnist-svm` model (e.g. input type, runtime to use, etc.).\n",
    "- `models/mushroom-xgboost/model-settings.json`: holds the configuration specific to our `mushroom-xgboost` model (e.g. input type, runtime to use, etc.).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `settings.json`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile settings.json\n",
    "{\n",
    "    \"debug\": \"true\"\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `models/mnist-svm/model-settings.json`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile models/mnist-svm/model-settings.json\n",
    "{\n",
    "    \"name\": \"mnist-svm\",\n",
    "    \"implementation\": \"mlserver_sklearn.SKLearnModel\",\n",
    "    \"parameters\": {\n",
    "        \"version\": \"v0.1.0\"\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `models/mushroom-xgboost/model-settings.json`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile models/mushroom-xgboost/model-settings.json\n",
    "{\n",
    "    \"name\": \"mushroom-xgboost\",\n",
    "    \"implementation\": \"mlserver_xgboost.XGBoostModel\",\n",
    "    \"parameters\": {\n",
    "        \"version\": \"v0.1.0\"\n",
    "    }\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start serving our model\n",
    "\n",
    "Now that we have our config in-place, we can start the server by running `mlserver start .`. This needs to either be ran from the same directory where our config files are or pointing to the folder where they are.\n",
    "\n",
    "```shell\n",
    "mlserver start .\n",
    "```\n",
    "\n",
    "Since this command will start the server and block the terminal, waiting for requests, this will need to be ran in the background on a separate terminal."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing\n",
    "\n",
    "By this point, we should have both our models getting served by MLServer.\n",
    "To make sure that everything is working as expected, let's send a request from each test set.\n",
    "\n",
    "For that, we can use the Python types that the `mlserver` package provides out of box, or we can build our request manually."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing our `mnist-svm` model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "x_0 = X_test_digits[0:1]\n",
    "inference_request = {\n",
    "    \"inputs\": [\n",
    "        {\n",
    "          \"name\": \"predict\",\n",
    "          \"shape\": x_0.shape,\n",
    "          \"datatype\": \"FP32\",\n",
    "          \"data\": x_0.tolist()\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "endpoint = \"http://localhost:8080/v2/models/mnist-svm/versions/v0.1.0/infer\"\n",
    "response = requests.post(endpoint, json=inference_request)\n",
    "\n",
    "response.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing our `mushroom-xgboost` model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "x_0 = X_test_agar[0:1]\n",
    "inference_request = {\n",
    "    \"inputs\": [\n",
    "        {\n",
    "          \"name\": \"predict\",\n",
    "          \"shape\": x_0.shape,\n",
    "          \"datatype\": \"FP32\",\n",
    "          \"data\": x_0.toarray().tolist()\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "endpoint = \"http://localhost:8080/v2/models/mushroom-xgboost/versions/v0.1.0/infer\"\n",
    "response = requests.post(endpoint, json=inference_request)\n",
    "\n",
    "response.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
